A Simulation Framework for Evaluating Designs for Sponsored Search Markets

Sponsored search is a rapidly growing business and there is tremendous industry and research interest in improving the designs and functioning of the sponsored search marketplace. Launching new designs and enhanced features for the sponsored search marketplace requires careful evaluation of their potential consequences to user experience and financial impact on the multiple parties (advertisers, publishers and marketplace operator) involved. The complexity of market dynamics makes it difficult to draw definite conclusions about the market without comprehensive testing. While limited field testing is often performed, it has several disadvantages: limited control over design parameters, limited sample sizes and scenarios that can be tested. Simulation testing is a viable option. Though some previous works have reported on the use of simulations, most of these are ad hoc and intended to test specific scenarios. In this paper, we describe the design of a general purpose simulation framework that supports the evaluation of alternative designs and features. We initially discuss the functional and architectural requirements for implementation of this framework. From a methodological perspective, there is a need to simulate a "micro-market" – a small scale representation of a complete market – for effective evaluation. Hence, we next describe how micro-market data samples are generated and an approach to scaling the metrics produced from simulations, using such samples, to represent an entire market. Finally, we relate our experiences in applying this simulation framework to evaluating designs and features for sponsored search at Yahoo!